1. Field of the Invention
The present invention relates to an image processing apparatus for processing a digital image signal, specifically for processing a luminance signal and chrominance signals of the digital image signal by using respectively different interpolation techniques.
2. Description of the Related Art
Computer graphics and image processing occasionally require geometric transformation of images, such as enlargement and reduction, rotation and deformation. Geometric transformation of vector-type data requires only coordinate transformation of each vector, but geometric transformation of raster images requires other processing in addition to coordinate transformation. Geometric transformation of raster images refers to projection of an existing image to a different coordinate systems based on a given coordinate transformation equation, and requires the following three stages of processing.
Based on a given coordinate transformation equation, an image coordinate of an input image is transformed into an image coordinate of an output image, or an image coordinate of an output image is transformed into an image coordinate of an input image.
Image data of the input image is rearranged so as to correspond to a lattice arrangement of the output image obtained by the coordinate transformation.
Image data of pixels (at the intersections of the lattice) which is required for rearrangement is obtained by interpolation. This is performed since post-coordinate transformation image data is not arranged in a lattice in general.
The rearrangement of image data will be described in more detail. Rearrangement of image data can be performed in two different methods. One method is based on the concept of forward transformation. By the concept of forward transformation, the position on the post-coordinate transformation output image coordinate system which corresponds the position of each pixel of an input image is calculated, and the image data of each pixel of the input image is projected onto the calculated position on the post-coordinate transformation output image coordinate system. The other method is based on the concept of reverse transformation. By the concept of reverse transformation, the position on an input image coordinate system which corresponds the position of each pixel of an output image is calculated, and the image data of the calculated position is obtained. Geometric transformations of raster images generally adopt the concept of reverse transformation. In either method, the coordinate of the position obtained by either method is not an integer in general, and thus interpolation is required.
2-1. Method Based on the Concept of Forward Transformation
The position on the post-coordinate transformation output image coordinate system which corresponds the position of each pixel of an input image is calculated. The image data of each pixel of the input image is projected onto the calculated position on the post-coordinate transformation output image coordinate system. Interpolation processing is performed on an output image coordinate system. The image data of each of pixels on the output image is obtained based on the image data of each pixel of the projected input image.
2-2. Method Based on the Concept of Reverse Transformation
The position on an input image coordinate system which corresponds the position of each pixel of an output image is calculated. The image data of the calculated position is obtained. Interpolation processing is performed on the input image coordinate system. The image data of each pixel of the output image is obtained based on the image data of each of the pixels of the input image arranged in a lattice.
In accordance with geometric transformation of raster images, a forward lattice is set on the post-coordinate transformation output image coordinate system, and the input image is transformed into the arrangement of the image data corresponding to the pixels. Whether the method based on the concept of forward transformation or the method based on the concept of reverse transformation is adopted, the coordinate of the corresponding position obtained by calculation (i.e., the coordinate in the output image in the case of forward transformation, and the coordinate in the input image in the case of reverse transformation) is not an integer in general. Accordingly, the image data of the pixel to be interpolated needs to be obtained by interpolation from the image data of other pixels in the vicinity of the pixel to be interpolated.
There are many interpolation techniques usable for geometric transformation of raster images. Hereinafter, the three main techniques (i.e., nearest neighbor interpolation, bi-linear interpolation, and cubic convolution interpolation) will be generally described.
Herein, the three main techniques will be described when rearrangement is performed based on the reverse transformation, which is common to geometric transformations of raster images. In the following description, the input image coordinate of the pixel to be interpolated is represented by (u, v); and the image data of the pixel is represented by P. A pixel having a pixel number of i and a line number of j is represented by (i,j), and the image data of an input image at pixel (i,j) is represented by Pi,j. Symbol [ ] is a Gaussian symbol and represents that decimals are omitted.
3-1. Nearest Neighbor Interpolation
The image data at pixel Pi,j, which is closest to pixel P to be interpolated is used as the image data of pixel P. The image data at pixel Pi,j is obtained by expression (1).
P=Pi,jxe2x80x83xe2x80x83(1)
where i=[u+0.5], j=[v+0.5].
This technique generates a positional error of xc2xd pixel at the maximum, but has advantages that the original data is not destroyed and the processing algorithm is simple.
3-2. Bi-linear Interpolation
The image data of pixel P to be interpolated is obtained using image data of four (2xc3x972) pixels (Pi,j, Pi,j+1, Pi+1,j, Pi+1,j+1) in the vicinity of pixel P based on expression (2).
xe2x80x83P=[(i+1)xe2x88x92u][(j+1)xe2x88x92v]Pi,j+[(i+1)xe2x88x92u][vxe2x88x92j]Pi,j+1+[uxe2x88x92i][(j+1)xe2x88x92v]Pi+1,j+[uxe2x88x92i][vxe2x88x92j]Pi+1,j+1xe2x80x83xe2x80x83(2)
where i=[u], j=[v]
This technique disadvantageously destroys the original data but advantageously obtains the effect of smoothing due to averaging.
3-3. Cubic Convolution Interpolation
The image data of pixel P to be interpolated is obtained using image data of 16 (4xc3x974) pixels (P1,1, P1,2, . . . P4,4) in the vicinity of pixel P, using the cubic convolution function represented by the expression (3).                     P        =                                            [                                                f                  ⁡                                      (                    y1                    )                                                  ⁢                                  f                  ⁡                                      (                    y2                    )                                                  ⁢                                  f                  ⁡                                      (                    y3                    )                                                  ⁢                                  f                  ⁡                                      (                    y4                    )                                                              ]                        ⁡                          [                                                                                          P                                              1                        ,                        1                                                                                                                        P                                              2                        ,                        1                                                                                                                        P                                              3                        ,                        1                                                                                                                        P                                              4                        ,                        1                                                                                                                                                        P                                              1                        ,                        2                                                                                                                        P                                              2                        ,                        2                                                                                                                        P                                              3                        ,                        2                                                                                                                        P                                              4                        ,                        2                                                                                                                                                        P                                              1                        ,                        3                                                                                                                        P                                              2                        ,                        3                                                                                                                        P                                              3                        ,                        3                                                                                                                        P                                              4                        ,                        3                                                                                                                                                        P                                              1                        ,                        4                                                                                                                        P                                              2                        ,                        4                                                                                                                        P                                              3                        ,                        4                                                                                                                        P                                              4                        ,                        4                                                                                                        ]                                ⁢                      xe2x80x83                    [                                                                      f                  ⁡                                      (                    x1                    )                                                                                                                        f                  ⁡                                      (                    x2                    )                                                                                                                        f                  ⁡                                      (                    x3                    )                                                                                                                        f                  ⁡                                      (                    x4                    )                                                                                ]                                    (        3        )            
Conventional interpolation techniques used for enlarging and reducing images include, for example, a digital image interpolation circuit for performing a plurality of interpolation techniques disclosed by Japanese Laid-Open Publication No. 8-251400. It is not described in the document that the plurality of interpolation techniques are used in parallel for one color image.
Japanese Laid-Open Publication No. 1-142879 discloses that the cubic convolution interpolation is used only for a luminance signal separated from a video signal source, but does not describe anything on processing of chrominance signals.
The above-described cubic convolution interpolation processes the data as follows. Pixels in the vicinity of the pixel to be interpolated are each multiplied with a positive coefficient, and the resultant values are added together, thus to smooth the image. Pixels far from the pixel to be interpolated are each multiplied with a negative coefficient, and the resultant values are added together; i.e., subtraction is performed. Thus, the difference between the value of the pixels in the vicinity of the pixel to be interpolated and the value of the pixels far from the pixel to be interpolated is emphasized. Accordingly, the cubic convolution interpolation has an advantage of smoothing and also sharpening the image, but has a disadvantage of requiring a large amount of calculation.
When, for example, the cubic convolution interpolation is applied for each of R, G and B components of a color image, the processing time period and the memory quantity used are three times larger than in the case of a monochrome image. This is a serious problem to solve in order to apply this technique to mobile devices such as digital cameras.
When the cubic convolution interpolation is applied for chrominance signals, the following problems arise. Since the operation area is as large as 4xc3x974 pixels, the result is influenced by pixels not in the vicinity of the pixel to be interpolated. Since there is a high possibility that pixels not close to one another represent different colors, different colors may be undesirably mixed. In addition, the cubic convolution interpolation involves differential subtraction for emphasizing the difference in pixel values. Thus, a slight difference in color is emphasized. This leads to a significant change in the ratio among the color components and thus amplifies the unintended color deviation.
In one aspect of the present invention, an image processing apparatus includes a signal processing section for processing an image signal including a first signal and a second signal. The signal processing section processes the first signal and the second signal using respectively different interpolation programs.
In another aspect of the present invention, an image processing apparatus includes a signal processing section for performing signal processing of an image signal including a first signal and a second signal; a program storing section having a plurality of types of interpolation programs stored therein; and a program switching section for selecting one of a first program for processing the first signal and a second program for processing the second signal among the plurality of types of interpolation programs, based on at least one parameter. The signal processing section includes a first signal processing section for processing the first signal using the first program and a second signal processing section for processing the second signal using the second program.
In one embodiment of the invention, the first signal includes a luminance signal, and the second signal includes a chrominance signal.
In one embodiment of the invention, the first program includes a program for executing interpolation in an operation area of 4xc3x974 pixels or larger, and the second program includes a program for executing interpolation in an operation area of 2xc3x972 pixels or smaller.
In one embodiment of the invention, the first program includes a program for executing cubic convolution interpolation, and the second program includes a program for executing at least one of bi-linear interpolation, nearest neighbor interpolation, simple subsampling, and simple interpolation.
In one embodiment of the invention, the program storing section includes a data table which stores a prescribed operation result obtained by the interpolation program.
In one embodiment of the invention, the at least one parameter includes a parameter representing a requirement for one of image quality, processing speed, and power consumption.
In one embodiment of the invention, the signal processing includes enlargement and reduction of the image signal.
In one embodiment of the invention, the image processing apparatus further includes an image input section for receiving the image signal and outputting the first signal and the second signal.
In one embodiment of the invention, the image processing apparatus further includes an image output section for synthesizing the first signal processed by the first signal processing section and the second signal processed by the second signal processing section and outputting the synthesized signal.
In one embodiment of the invention, the image input section includes a CCD camera.
In one embodiment of the invention, the image output section includes a liquid crystal display.
In one embodiment of the invention, the image signal includes a signal representing a still picture.
In one embodiment of the invention, the image signal includes a signal representing a moving picture.
According to the present invention, the signal processing section processes a luminance signal and chrominance signals of an image signal by using respectively different interpolation techniques.
The human eye is more sensitive to the luminance signal than to the chrominance signal. Accordingly, an interpolation technique for providing a high image quality is applied for the luminance signal, and an interpolation technique requiring a small amount of calculation is applied to the chrominance signals. In this manner, the amount of calculation can be reduced thus shortening the data processing time period while maintaining the high image quality. In addition, since the chrominance signals can be represented by a prescribed format (422 system, 411 system, etc.) requiring a smaller amount of data than the data amount of the original color components, the quantity of a memory required for storing an intermediate result can be reduced.
In the above, xe2x80x9c422xe2x80x9d and xe2x80x9c411xe2x80x9d represent the ratio of the amount of information assigned to different signals in the YCrCb system. In the 422 system, information is assigned at the ratio of Y:Cr:Cb=4:2:2. In the 411 system, information is assigned at the ratio of Y:Cr:Cb=4:1:1. In other words, the amount of information is compressed in order to reduce the required quantity of memory, utilizing the fact that the human eye is less sensitive to color difference than to luminance difference.
In an embodiment where an interpolation technique performed in a small operation area is applied for the chrominance signals, the following effects are provided. The color difference, which is otherwise generated due to the distance between the pixels used for interpolation, is less likely generated. Since differential operations are not involved, color deviation can be avoided.
Therefore, the data processing time period can be shortened and the quantity of memory used can be reduced. In addition, an image fulfilling the requirements regarding image quality, processing speed, power consumption and the like, with substantially no color deviation, can be provided.
The above-described effect is achieved by, for example, applying an interpolation performed in an operation area of 4xc3x974 pixels or more for the luminance signal and applying an interpolation performed in an operation area of 2xc3x972 pixels or less for the chrominance signals.
The above-described effect is achieved by, for example, applying cubic convolution interpolation for the luminance signal and applying bi-linear interpolation, nearest neighbor interpolation, simple subsampling, or simple interpolation for the chrominance signals.
The image processing apparatus according to the present invention can include a program storing section having a plurality of types of interpolation programs stored therein, a program switching section for selecting one of interpolation programs for performing signal processing in accordance with the requirements, and a signal processing section for performing signal processing using different interpolation techniques for a luminance signal and chrominance signals of a digital image signal.
Therefore, an optimum program can be selected among a plurality of types of interpolation programs stored in the program storing section in accordance with the requirements regarding image quality, processing speed, power consumption and the like; and the signal processing section can process the luminance signal and the chrominance signals using different interpolation techniques.
The image processing apparatus according to the present invention can be applied to, for example, a digital still camera having a moving picture shooting function. A main function of a digital still camera is shooting of a still picture. A high image quality with a great number of pixels is required for a still picture. For a moving picture, each of images of a moving picture has a small number of pixels since the amount of data which can be transferred per a unit time is limited due to, for example, the limitation on the capacity of the memory of the digital still camera (i.e., limitation of the shooting time period) and the limitation on the band width of the data transfer path. Nor is such a high image quality required.
Accordingly, various interpolation techniques can be used as follows. During digital zooming when shooting a still picture, cubic convolution interpolation is applied for the luminance signal and a simpler interpolation technique is applied for the chrominance signals. During digital zooming when shooting a moving picture, bi-linear interpolation is, for example, applied for the luminance signal and a simpler interpolation technique is applied for the chrominance signals.
Thus, an image fulfilling the requirements regarding image quality, processing speed, power consumption and the like, with substantially no color deviation, can be provided.
In an embodiment where a data table having a prescribed operation result of the interpolation program is used as the interpolation program, the amount of operation can be reduced and the data processing speed is further raised.
Thus, the invention described herein makes possible the advantages of providing an image processing apparatus for processing a digital image signal in a shorter period of time with a smaller quantity of memory while fulfilling specific requirements (parameters) regarding image quality, processing speed, power consumption and the like, and thus providing an image with substantially no color deviation.
These and other advantages of the present invention will become apparent to those skilled in the art upon reading and understanding the following detailed description with reference to the accompanying figures.